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^ (57) Abstract: The image processor (101) for motion compensated image processing comprises a motion compensation unit (106) 
^ for calculating an output image based on a particular motion vector field and based on a farst input image and a second input image. 
^ The particular motion vector field can be calculated by a motion estimator (104) which is part of the image processor (101) or by 

an external unit (108). The image processor (101) is designed to select from which source, i.e. the motion estimator (104) or the 
O external unit (108), the particular motion vector field is taken to calculate the output image. The external source is designed to 

calculate the particular motion vector field based on a motion vector field which was provided by the motion estimator (104) of the 
>^ image processor (101). 
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Image processor and image display ^paratus provided witii sucli processor 



The invention relates to an image processor for motion compensated image 

processing, comprising: 

- a memory unit for storing a first input image and a second input image; 

- a motion estimator for estimating a motion vector field based on the first 
5 iapvA image and the second input image; and 

- a motion compensation unit for calculating an ouQ)ut image based on a finrther motion 
vectOT field and based on tiie first inpat image and the second ir^ut image. 

The invraition fiirther relates to an image disqplay apparatus comprising: 

- a receiver for receiving a sequence of images conq>rising a first input image 
10 and a second output image; 

- an image processor for motion compensated image processing, con^sing: 

- a memory unit for storing the first iapxxt image and tiie second inpat image; 

- a motion estimator for estimating a motion vector field based on tiie first 
input image and tide second input imag^ and 

15 -a motion compensation unit for calculating an output image based on a 

fiffttier motion vector field and based on the first input image and tiie second inptA image; 

- a display device for displaying tihie output image. 



20 An embodiment of tiie image processor of tihte kind desaSbed in the opening 

paragraph is known fiom the article 'IC for motion-con5>einsated de-interlacing, noise 
reduction, and picture rate conversion", by G. de Haan, in IEEE Transactions on Consumer 
Electronics, Vol. 45, No. 3, August 1999. 

hnage sequences come in various image rates: fihn material in 24, 25 and 30 

25 Hz and video usually in 50 Hz and 60 Hz. Television displays, on the otiier hand, are 

commercially available with image rates of 50, 60 and 100 Hz. and have either progressive o 
interlaced scanning. Simple image rate converters repeat images until the next image arrives, 
which results in blur and/or judder when motion occurs. Similarly, de-interlacing sometimes 
results fiom repetition, or averaging of neighboring lines. The more advanced de-intrarlacing 
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concepts apply vertical-temporal processing, but even these degrade those portions of images 
where motion occurs. 

In the cited article it is described that consumer television ICs appeared in the 
mid-nineties using motion estimation and motion compensation to achieve high performance 
5 conversion for ev^ moving sequences. The article shows improved algorithms for motion 
estimation, motion compensated de-interlacing, motion compensated image interpolation and 
motion corrqiensated noise reduction. The IC implementing these algorithms comprises a 
memory imit to cache a current and a previous image, a motion estimator and a motion 
compensation unit The working of the IC is as follows. Two consecutive input images are 

1 0 received and cached in the memory unit of the IC. Based on these input images motion 
vectors are calculated. The set of motion vectors of one image pair, i.e. a current and a 
previous image, is called a motion vector field. The motion vectois are provided to the 
motion compensation unit which performs the operations as listed above. The result is a 
motion compensated oulput image. The IC performs the operations in real-time. This means 

15 that a motion estimation algorithm witii a relatively low operation count is implemented. 

Although flie results of the motion estimation unit are relatively good, there can be erroneous 
motion vectors, resulting in artifacts in the motion compensated images. Especially at the 
borders of the images or at occlusion areas ki the images. More advanced motion estimation 
is required in these cases. 

20 



It is a first object of the invention to provide an image processor of the kind 
described in the opening paragraph that is designed to enable more advanced motion 
estimation. 

25 It is a second object of the invention to provide an image display apparatus of 

the kind described in the opening paragr^h with more advanced motion estimation. 

The first object of the invention is achieved in that the image processor is 
designed to selectively receive the further motion vector field from: 

- the motion estimator; or firom 
30 - an extCTial unit 

The main difference with the image processor of the prior art is that the 
motion vectors can be calculated by the motion estimator of the image processor itself but 
that the motion vectors can also be calculated by an external unit, i.e, a source not being part 
of the processor itself, which is arranged to act as co-proc^or. It is preferred that this co- 
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processor is designed to perform more advanced motion estimation than the motion 
estimation of the motion estimator. Notice that the further motion vector field may 
correspond to the motion vector field calculated by the motion estimator but that it also may 
be a motion vector field which has been provided by the external unit 

An embodiment of the image processor according to the invention is designed 
to provide the motion vector field to the external unit which is arranged to calculate the 
jSirther motion vector field on the basis of the motion vector field. In this embodiment the 
motion vectors can be exchanged to the external unit which is arranged to act as co-processor 
to inqoove tiie motion vector field which is calculated by the motion estimator of the image 
) processor. The advantage is that the available motion estimator of the image processor is 
used and the resulted motion vector field is improved by the rattemal unit The image 
processor has two modes of operations: 

- One pass mode: The first mpat image and the second input image are 
received and stared in the memory unit of the image processor. Motion vectors are calculated 

15 based on these input images. These motion vectors are provided to the motion compensation 
unit which calculates the moticm compaisated output image. 

- Two pass mode: In the first pass of the two passes, the first inpvt image and 
the second iiq)ut image are received and stored in the memory unit. Motion vectors are 
calculated based on these h^ut images. These motion vectors are provided to the external 

20 unit. Externally the received motion vector field can be tenqwrarily stored. But most 

important, the external unit is arranged to perform operations on Uie motion vector field in 
order to improve it. In the mean time Ihe image processor can continue with processing other 
hnages. E.g. tiie motion compensation unit of the image processor can calculate a second 
motion compensated output hnage or tiie motion estiimalor of the image processor can 
25 calculate a motion vector field of another image pair. In the second pass of the two passes, 
the fiirther motion vector field corresponding to the first and second input image are provided 
to the motion compensation unit together with the first and second input image. Based on this 
inptit the output image is calculated. 

In the two pass mode the first and the second input image have to be read mto 
30 the memory unit twice: once for the first pass and once for the second pass. The result is an 
additional load of tiie memory bus. However the main advantage of the image processor 
according to the invention which is designed to alternate between the one pass and the two 
pass mode is the scalability. It allows to choose betwem an increased image quaUty at the 
cost of some additional bandwidth. 
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In another embodiment of the image processor according to the invention, in 
which the motion vector field is provided to the external imit, the external unit is arranged to 
recalculate a particular motion vector of the motion vector field. Even more advanced 
adaptations of the motion vector field can be applied. Kg. by using extended sets of 
5 candidate motion vectors to find an appropriate motion vector. Motion estimation based on 
more than two consecutive images also belongs to the possibilities. 

An embodiment of the image processor according to the invOTtion, in which 
the motion vector field is provided to the external unit, comprises a switch with: 

- a first input connector that is coupled to the motion estunatoi^ 
10 - a second input coimector that is coupled to the external unit; and 

- an output connector that is coupled to the motion comqpensation unit, 
the switch having: 

- a first state in which the first input connector is coimected to the output 

connector; and 

15 - a second state in which the second input connector is connected to the output 

coimector. 

The advantage of the switch is that it makes the design of tiie motion 
con:q)ensation unit relatively easy. The motion compensation unit comprises one interface to 
which with the appropriate motion vector fields are provided in stead of two interfaces which 

20 potentially can provide the further motion vector field. 

In an embodiment of the image processor according to the invention, in which 
the motion vector field is provided to the extemal unit, the motion estimator comprises a first 
control interface in order to enable and/or disable the motion estimator. The advantage of the 
first control interface is that it makes the design of the motion estimator unit relatively easy. 

25 A maia part of the timing and control of the motion estimator is executed extemally to the 
motion estimator. 

In an embodiment of the image processor according to the invention, in which 
the motion vector field is provided to tiie extemal unit, the motion compensation unit 
comprises a second control interface in order to enable and/or disable the motion 
30 compensation unit. The advantage of the second control interface is that it makes the design 
of the motion compensation unit relatively easy. A main part of the timing and control of tiie 
motion compensation imit is executed extemally to the motion compensation unit 

In an embodiment of the image processor according to the invention the 
motion compensation unit is designed to perform at least one of the following operations: 
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- De-interlacing: Common video broadcast signals are intedaced, which 
impUes that successive images contain only the even or only the odd lines alternately. De- 
interlacing attempts to restore the full vertical resolution, i.e. make both the odd and even 
lin^ available for each image; 

- Temporal image interpolation: From a series of original mput images a larger 
series of output images is calculated. Output images are temporally located between two 

original input images; and 

- Temporal noise reduction. This can also involve spatial processing, resulting 

in spatial-tonporal noise reduction. 

The second object of the invention is achieved in that the image processor of 
fb& image display ^aratus is designed to select a source from a set of sources, the source 
providing the fuifha- motion vector field to the motion compaisation unit and the set of 
sources comprising: 

- the motion estimator; and 

- an external unit to which the motion vector field is provided. 
Modifications of the image processor and variations thereof may correspond to 

modifications and variations thereof of the image display apparatus described. 



These and oliier aspects of the image processor and of ttie image display 
apparatus according to fee invention will become jq>parent fi»m and wiU be elucidated with 
respect to the implementations and embodiments described hereinafter and with reference to 
the accompanying drawings, wherein: 

Fig. 1 A schematically shows an embodiment of the image processor in its 

context; 

Fig. IB schematically shows an embodiment of the image processor 
comprising a switoh to select between motion vectors from the motion estitnator or from the 
external unit; 

Fig. 2 schematically shows the data flow of the motion vector field in an 
embodiment of the im^e processor running in the one pass mode; 

Fig. 3A schematicaUy shows the data flow of the motion vector field in an 
embodiment of the image processor during the first pass of the two pass mode; 

Fig. 3B schematically shows the data flow of the further motion vector field in 
an embodiment of Ihe image processor during the second pass of the two pass mode; and 
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Fig. 4 schematically shows an image display ^aratus according to the 

invention. 

Corresponding reference numerals have the same meaning in all of the Figures. 



Fig. 1 A schematically shows an embodiment of the image processor 100 in its 
context The image processor 100 comprises: 

- a memory unit 102 for storing a first input image and a second input image; 

- a motion estimator 104 for estimating a motion vector field based on the Gist 
10 input image and the second input image; and 

- a motion compensation unit 106 for calculating an output image based on a 
fiirther motion vector field and based on the first input image and the second input image. 

The image processor 100 is designed to select a source firam a set of sources, 
the source providing the fiirther motion vector field to the motion compoisation unit 106 and 
15 the set of sources comprising: 

- the motion estimator 104; and 

- an external unit 108. 

The data bus 1 12 is designed to exchange e.g. data of input and output images 
and motion vectors. To the data bus 1 12 are connected: 
20 - the image processor 100; 

- an external memory device 110 arranged to store e.g. data of input and 
output images and motion vectors; and 

- the extonal unit 108. 

The woridng of the image processor is as follows. A first and a second input 
25 image are retrieved firom the external memory device 1 10 via the data bus 1 12 and stored in 

the memoiy-unit 1 Q2..Itis possiblethat alLpixels-of the first and.secondJnput image are . „ 

stored in the memory unit 102 simultaneously. But preferably only a portion of Ifaese pixels is 
simtdtaneously stored in the memory unit 102. The fiorst and the second input image are 
exchanged via image data path 116 and image data path 118, respectively. In the case that no 
30 motion vector field of this image pair has been calculated before, the first and the second 
input image are provided to the motion estimator 104. The motion estimator 104 calculates 
the motion vector field and provides this to the motion compensation unit 106 via motion 
vector path 1 14. The first and the second input image are also provided via the image data 
paths 126 and 128 to the motion compensation unit 106, which calculates the corresponding 
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output image. In the case that a motion vector field of this image pair has been calculated 
before the further motion vector field is retrieved firom the external source 108 and provided 
to the motion compensation unit 106 via motion vector path 120 together with the first and 
the second input image. Based on this input the motion compensation unit 106 calculates the 
corresponding output image. The output image is exchanged via image data path 1 15. Image 
data is provided to the external unit 108 via image data path 125 and motion vectors are 
retrieved fiom the external unit 108 via motion vector path 123. The motion compensation 
unit 106 is designed to perform at least one of the following operations: 

- de-interlacing; 

- tCToporal image interpo^on; or 

- tonporal noise reduction. 

Preferably the motion estimator 104 and of the motion compeosation unit 106 
operate according to algorithms as described ia the article 'IC for motion-compensated de- 
interiadng, noise reduction, and picture rate conversion", by G. de Haan, in IEEE 
Transactions on Consamer Electronics, Vol. 45, No. 3, August 1999. Optionally tiie de- 
interlacing is perfonned in accordance with another method as described in 'T)e-interlacing- 
An Overview" by G. de Haan, in proceedings of the IHBE, Vol. 86, No. 9, Sq>tember 1998. 

It is preferred that the image processor is implemeaited on one IC. 
Alternatively, the image processor is implemented with multiple ICs which are 
intercoimected with connections which have a relatively krge bandwidth. 

Fig. IB schematically shows an embodiment of the image processor 101 

comprising a switch 130 witii: 

- a first mput connector that is coupled to the motion estimator 104 via motion 

vector path 114; 

- a second mput connector that is coupled to the extamal miit 108 via motion 

vector path 120; and 

- an output connector that is coupled to the motion compensation unit 106, 

and the switch (130) having: 

- a first state in which the first mput connector is connected to the output 
connector, enabling the further motion vector field to be provided directly firom the motion 
estimator 104 to the motion compensation unit 106; and 

- a second state in which the second input connector is connected to the output 
connector caiabling the further motion vector field to be provided firom tiie external unit 108 
to the motion compensation unit 106. 
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The motion estimator 104 comprises a first control interface 134 in order to 
enable and/or disable the motion estimator 104 and the motion compensation imit 106 
comprises a second control interface 132 in order to enable and/or disable the motion 
compensation unit 106. A main part of the timing and control of the motion estimator 104 
and of the motion compensation unit 106 is executed externally to these xxmts 104,106. 

The memory unit 102 comprises a first sub-unit 103 for storing the first input 
image and a second sub-unit 105 for storing tiie second input image. 

The image processor 101 is arranged to exclwaage the motion vector field firom 
the motion estimator 104 to the external unit 108 via the motion vector paths 1 13 respectively 
124. 

In connection with Fig. 2, Fig. 3A and Fig. 3B it will be described under 
which coiMiitions the switch is ia the first or the second state and whether the motion 
estimator 104 and motion compensation unit 106 are enabled or disabled. 
This is smnmarized in Table 1 . 
Table 1: 





One pass 
(see Fig. 2) 


First pass of two passes 
(see Fig. 3 A) 


Second pass of two passes 
(see Fig. 3B) 


motion estimator 104 


Enabled 


Enabled 


Disabled 


motion compensation 
unit 106 


Enabled 


Disabled 


Enabled 


Switch 130 


First state 


Second state 


Second state 



Fig. 2 schCTiatically shows the data flow 202 of the motion vector field in an 
embodiment of the image processor 101 running in the one pass mode. Both the motion 
estimator 104 and the motion compensation unit 106 are enabled as indicated with check 
marks 204 and 206. The switch 130 is in the first state. The sequence of processing steps is as 
follows. The first input image and the second input image are received &om the external 
memory device 1 10 and cached in the memory unit 102 of the image processor 101. The first 
image, i.e. the so-called previous image, is put in sub-unit 103 and the second input image, 
i.e. the so-called previous image is put iu sub-unit 105. The images are divided in multiple 
segments. For each of the segments amotion vector is calculated. All vectors of one image 
together form a motion vector field. These motion vectors are directly provided to the motion 
compensation unit 106, which calculates the motion compensated output image. In principle, 
the motion vectors are provided one by one or in small sets. But eventually the complete 
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motion vector field is provided. The advantage of providing the motion vectors one by one is 
that the motion compensation unit 106 can complete the output image relatively short after 
the last motion vector has been calculated. Another advantage is that no memory space is 
required in the image processor 101 to store the complete motion vector field. Typically the 
motion vectors are also stored in liie extCTnal memory device 110. 

Fig. 3A schranatically shows the data flow 302 of the motion vector field in an 
embodiment of the image processor 101 during tiie first pass of the two pass mode. The 
motion estimator 104 is aiabled and the motion compensation unit 106 is disabled as 
indicated with reference numerals 304 and 306. The switch 130 is in the second state. 
However the state of tiie switch 130 is not relevant because the motion ccrnqjcnsation unit 
106 is disabled. The sequence of processing stqps is as follows. The first input image and the 
second input image are received fcom tiie external memory device 1 10 and cached in tiie 
memory unit 102 of flie image processor 101. The images are divided in multiple segments. 
For each of tiie segments a motion vector is calculated. These motion vectors are temporarily 
stored in tiie external memory device 1 10. The extanal unit 108 can access tiiese motion 
vectors. The external unit 108 is designed to calculate a fiirfher motion vector field, 
optionally based on flie motion vectors being calculated by flie motion estimator 104. The 
further motion vector field calculated by tiie external unit 108 will be used by ttie image 
processor 101 in tiie second pass. Optionally tiiis fijrftier motion vector field is also stored in 

the external memory device 110. 

Figl 3B schematically shows tiie data flow 312 of tiie finrflier motion vector 
field in an embodiment of tiie image processor 101 during tiie second pass of tiie two pass 
mode. The motion estimator 104 is disabled and tiie motion compensation unit 106 is enabled 
as indicated witii reference numerals 308 and 310. The switch 130 is in tiie second state. The 
sequence of processing st^s is as follows. The first input image and flie second input image 
are received fiom tiie external memory device 1 10 and cached in tiie memory unit 102 of flie 
image processor 101 . The particular motion vectors corresponding to tiie first and second 
input image are provided by tiie extemal source 108 or retrieved firom flie external memory 
110. The motion compensation unit 106 calculates tiie output image. The scanning direction, 
i.e. tiie sequence of segments being processed, during flie second pass is independent of tiie 
scanning direction during flie first pass. In other words, during tiie first pass motion vectors 
can be calculated for segments fiom bottom-right to top-left of tiie image while during tiie 
second pass tiie motion compensated output image can be created by processing segments 
fiom e.g. top-left to bottom-right. This allows alternating tiie scanning directions of tiie 
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motion estimator, which increases the convergenge speed of the algorithms, while the motion 
compensation always appUes the same scanning direction. 

The external imit 108 and the image processor 101 are both designed to 
calculate motion vectors that are related to the same sequence of images. The image 
processor 101 is designed to calculate a first motion vector field of a particular image pair 
during a first time slot The extemal unit 108 is arranged to calculate a second motion vector 
field of the particular image pair during a second time slot based on the first motion vector 
field In a third time slot the second motion vector field will be used by the motion 
compensation unit 106 of the image processor 101 to calculate an output unage. However the 
extemal unit 108 and the image processor 101 are processors which run in parallel. The effect 
of flie parallel processing is illustrated with an example. 
The example is summarized in Table 2. 
Table 2: 



Time slot 


extemal unit 108 


Image processor 101 


motion estimator 104 


motion compmsation miit 106 


1 




F,iN,N+l) 




2 






F,iN-l,N) 


3 


F,iN + l,N + 2) 




4 






F^iN,N + l) 


5 


Fi(N+2,N+3) 




6 


F2(N + 2,N+3) 




F2iN + l,N + 2) 



15 During timeslot 1 the motion estimator 104 of the image processor 101 is 

calculating a motion vector field (N, iV + 1) of image pair N,N+1 while the extemal unit 
1 08 is calculating a motion vector field F2 (AT - 1, iV) of image pair N-1 ,N. During timeslot 2 
the motion compensation unit 106 of the image processor 101 is using the motion vector field 
F2 (N-l^N) of image pair N-1,N while the extemal unit 108 is calculating a motion vector 

20 field F2 (N, N + l) for image pair N,N+1 . Notice that in this example the extemal unit 108 

can spend more time on calculating a motion vector field than the image processor can spend. 
Taking also into accoxmt that in most cases only a portion of tiie motion vector field require 
an improvement then it is convincing that the two pass mode results in improved image 
quahty. 
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Fig. 4 schesmaticaUy shows an image display apparatus 400 according to the 

invention comprising: 

- a receiver 402 for receiving a sequence of images. The images may be 
broadcasted and received via an antenna or cable but may also come from a storage device 

5 like a VCR (Video Cassette Recorder) or DVD (Digital Versatile Disk). The sequence of 
images is provided at fte input connector 406 of the image display apparatus 400. Image 
sequ^ces come in various image rates: film material in 24, 25 and 30 Hz and video usually 

in 50 Hz and 60 Hz; 

- an image processor 101 implemented as described in connection with Fig. 

10 IB; and 

- a display device 404 for displaying images. The type of the display device 
404 may be e.g. a CRT, LCD or PDF. The display device 404 can operate with image rates of 
50, 60 or 100 Hz aoA can have either progressive or interlaced scanning. 

It should be noted that the above-mentioned embodiments ilhistrate rather than 
15 limit the invention and lhat ttiose skiUed in the art will be able to design alternative 

embodiments without d^arting from the scope of the appended claims. In the claims, any 
reference signs placed between parentheses shall not be constructed as limiting the claim. 
The word ' comprising' does not exclude the pteseaice of elements or steps not listed in a 
claim. The word "a" or "an" preceding an element does not exclude the presence of a 
20 pluraUty of such elements. The invention can be implemented by means of hardware 

comprising several distmct elements and by means of a suitable programmed computer. In 
the unit claims enumerating several means, several of these means can be embodied by one 
and the same item of hardware. 
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CLAIMS: 



1. Aa image processor (100,101) for motion compensated image processing, 
comprising: 

- a memory unit (102) for storing a fibrst input image and a second irput image; 

- a motion estimator (104) for estimating a motion vector jfield based on the 
5 j&rst input image and the second input image; and 

- a motion compensation unit (106) for calculating an output image based on a 
further motion vector field and based on the first input image and the second input image, 
characterized in being designed to selectively receive the fiirther motion vector field from: 

- the motion estimator (1 04); or from 
10 - an external unit (108). 

2. An image processor (100,101) as claimed in Claim 1, characterized in that the 
image processor (100,101) is designed to provide the motion vector field to the external unit 
(108) which is arranged to calculate the further motion vector field on the basis of the motion 

15 vector field. 

3- An image processor (100,101) as claimed in Claim 2, characterized in IMt the 

external unit (108) is arranged to recalculate a particular motion vector of the motion vector 
field. 

20 

4. An image processor (100,101) as claimed in Claim 2, characterized in 

comprising a switch (130) with: 

- a first input connector that is coupled to the motion estimator (104); 

- a second input coimector that is coupled to the external unit (108); and 

25 - an output comiector that is coupled to the motion compensation unit (106), 

the switch (130) having: 

- a first state in which the first input connector is connected to the output 
connector; and 
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- a second state in which the second input connector is connected to liie ou^tit 

connector. 

5. An image processor (100,101) as claimed in Claim 2, characterized in that the 
motion estimator (104) comprises a first control interface (134) in order to enable and/or 
disable the motion estimator (104). 

6. An image processor (100,101) as claimed in Claim 2, characterized in tiiat the 
motion compensation unit (106) comprises a second control interface (132) in order to enable 
and/or disable flie motion compensation unit (106). 

7. An image processor (100,101) as claimed in Claim 1, characterized in tiiat the 
motion compensation unit (106) is designed to perform at least one of flie foUowing 
operations: 

- de-interlacing; 

- temporal image interpolation; or 

- temporal noise reduction. 

8. An image display apparatus (400) comprising: 

- a receiver (402) for receiving a sequence of ims^es comprismg a first input 

image and a second output image; 

- an image processor (100,101) for motion compensated image processing, 

conqnising: 

* a memory unit (102) for storing the first input image and the second input 

image; 

* a motion estimator (104) for estimating a motion vector field based on the 
first iapat image and titie second input linage; and 

* a motion compensation unit (1 06) for calculating an output image based on a 
fiirflier motion vector field and based on the first input image and llie second input image; 

- a display device (404) for displaying the output image, characterized in that 
the image processor (100,101) is designed to selectively receive the fiirther motion vector 
field firom: 

- the motion estimator (104); or fi:om 

- an external unit (108). 
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9. An image display ^aratus (400) as claimed in Claim 8, characterized in that 
that the image processor (100,101) is designed to provide the motion vector field to the 
external unit (108) which is arranged to calculate the further motion vector field on the basis 
of the motion vector field. 

10. An image display apparatus (400) as claimed in Claim 8, characterized in that 
the motion compensation unit (106) is designed to perform at least one of the following 
operations: 

- de-interlacing; 

- temporal image interpolation; or 

- tCTaporal noise reduction. 
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